U.S. DEPARTMENT OF STATE 
THIRD ANNUAL CONFERENCE ON PROGRAM EVALUATION: 
NEW PARADIGMS FOR EVALUATING 
DIPLOMACY IN THE 2 1ST CENTURY 
June 8-9, 2010 



Cultural Diplomacy Track 



PowerPoint presentation from workshop on 
Electronic Media Engagement 

Session transcript : 

http: //www. state . gov/s/d/rm/rls/rm/2010/148058 . htm 



Electronic Media Engagement 
Evaluation Design and Methodology 

Evaluation and Measurement Unit (EMU) 
Office of Policy, Planning and Resources (R/PPR) for 
Public Diplomacy and Public Affairs 



EVALUATION & MEASUREMENT UNI 




EVALUATION & MEASUREMENT UNIT 




Introduction 
Cherreka Montgomery 
Director, R/PPR EMU 



2 



Evaluation & measurement Unit 



Electronic Media Engagement Evaluation 

• Purpose : Respond to GAO and fulfill 2008 IIP request to 
investigate if and how electronic media engagement increase 
knowledge of U.S. policies, society and values 

• Goal : Conduct Market Research in Select Countries that 
identifies foreign public interest to the Department of State 

• Goal : Assess Posts' processes for communicating with foreign 
publics over the Internet given local constraints 

• Goal : Provide recommendations for process improvements at 
Post and IIP level 
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Evaluation & measurement Unit 




Operational Definition of Electronic Media 

Web 2.0 technologies and electronic media outreach 
tools that facilitate direct engagement with target 
audience members via 

- Information websites such as America.gov 

- Blogs - websites that feature regular entries of 
commentary, descriptions if events, or other material 
such as graphics of video (such as blog pages on 
America.gov) 

- Social networking sites (e.g. eJournal USA on 
Facebook) 

- Multimedia platforms (webchats, podcasts, RSS 
feeds, SMS feeds, YouTube) 



America nov 



facebook 



twitter 



YoujEEl 
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Evaluation & measurement Unit 




Goal 1: Conduct Market Research in Select Countries that 
identifies foreign public interest to the Department of State 

Juliet Dulles 
Social Science Analyst 
R/PPR EMU 
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Evaluation ent Unit ^R!: Logic Model: Goal 1 





Inputs 


Process 


Process Outcomes 


Ultimate Outcomes 


Market Research 


Web Conversations 
about Topics of 
Interest to the DoS 


1. Conversation 
Monitoring across 10 
topics in six 
languages (plus 
English) 

2. Classification of 
conversations 
according to tone 
and sentiment 
(initially in English 
only) 

3. Tracking of topic 
interest around 
planned events (pre- 
and post-test model) 


1. Tracking topical conversations 
overtime 

2. Measurement of the impact of 
events on conversational 
quantity, tone, and sentiment 

3. Measurements of the influence 
of IIP interventions in influencing 
conversational quantity, tone, and 
sentiment 


1. Ability to focus IIP outreach 
activities when and where needed 

2. Ability to identify how 
conversations about DoS issues 
changes over time 


Adaption and usage of Web 

"} n a nnl i/-atir»nc arrncc anrl 
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between countries (8) and 
cultures (several) 


1. Web survey of Internet users 

2. Tracking of site visits, 

infr»rrr»atir»n \/io\A/c anrl 
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downloads from US websites 

3. Intentional and scheduled IIP 
interventions 


1. User profiles 

2. Web 2.0 tool usage 

3 1 irrcj react i n I IQ t/inirc 

3. INlclcbl III Uj LUpiLb 

4. Engagement in US-provided media 

5. Success of IIP to influence knowledge-seeking 
behavior 


1. Identification of preferred Web 2.0 interaction 
modes by sub-group 

Z. lUtMIUI ILdllUll Ul LCjpiL IIILclfcibLb Uy bUU-glUUp 

3. Measurement of success of IIP in influencing 
usage of Us-provided material 


In-person interviews in of 
Internet users in Pakistan 


TBD 


TBD 


TBD 


Formative 
Evaluation 


1. IIP Mission, Goals, and 
Objectives 

2. Review of IIP policies and 
procedures 

3. Review of IIP products 
and services, strategies and 
tactics 


1. Interviews, Document Review 

2. Compile policies and 
procedures 

3. Compile product and services 
inventory 

4. Document IIP strategies and 
tactics 


1. Unified statement of Mission, Goals and 
Objectives 

2. Measurement Needs and Metrics 

3. Policy review and recommendations 

4. Identification of Topics of Interest 

5. Comparison of products and services to industry 
standards 

6. Assessment of strategies and tactics in relation 
to goals and objectives 


1. Assessment of the relationships between mission, 
goals and objectives AND policies and procedures 
AND products and services AND strategies and 
tactics 

2. Recommendations for refinements in any of the 
above 

3. Implementation of a measurement methodology 

4. Measurement over time of changes 
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Evaluation Hot 

/ 

Technology Partners 

• Discuss purpose of studying blogosphere as complement to 
traditional tools (focus groups, surveys) 

• Morningside Analytics: In & Out Links 

- Identifies groups of people who are talking/listening with 
each other 

- Example on following slide of the Arabic blogosphere 

• Crimson Hexagon: Tone and sentiment analysis 

- Identifies statistical patterns in the words used to express 
opinions on different topics 

• Morningside Maps, Crimson Analyzes the Conversation 



Evaluation uium ent UNirSffr 

/ 

Topic Creation Methodology 

• Topic Creation: Information From Post, Guidance based 
Morningside Process, Seed URLs, Develop Map of 
conversations 

• Tone & Sentiment 

- Requires interpretation by people with knowledge of 
language and culture 

- Language specific, not necessarily country specific 

• Dialects complicate interpretation 



Group: Initial Category Framework 




Positive Themes 




Negative Themes 



Relevant Categories 



Enjoys 
American 
pop-culture 
(movies, 
music, etc.) 


Appreciates 
American 
hospitality/ 
openness 


America the 

beautiful/ 

tourism 


Excited by 

American 

innovation 


Positively 
surprised by / 
impressed by 
America 


Dislikes / 
disapproves 
of American 
pop-culture 


American 
extremism / 
fanaticism 


Too 

materialistic 


American 
technological 
/ corporate 
imperialism 


Negatively 
surprised / 
disappointed 
by America 



Hidden Categories: 

• Off -topic /NA 
Spam 



crimson hexagon 
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Group: Iterate to Improve Classification 
Framework 




Positive Themes 




Negative Themes 



Enjoys 
American 
pop-culture 
(movies, 
music, etc.) 



Dislikes / 
disapproves 
of American 
pop-culture 



Expanded to 

include 
character/ 
< society 



American 
extremism / 
fanatici r 



America the 
oeautiful / 



.unsm 



Excited by 
American 
innovHtio r 
| Expanded 
I to include 
products 



Too 

materialistic 
Merged to include 
character / society 



American 
technological 
/ corporate 
sialism 




Added: 



crimson hexagon 



Negative opinion of American government 
Mixed opinion of America 
Teasing America 
Facts 
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Evaluation 

j 

Deep Dive: Trends Over Time in Relevant Categories 



What Opinions about America are Expressed in 
Russian Link Economy? 



Reset Results 


Sharing 


Alerts 


Edit 



Creator: Ed Schneider on Mar 9, 2010 



Results Overview 

All Categories 



Content Sources Sample Posts 



Result Summary from Feb 15, 2010 to May 20, 2010 



Analysis Status 



Trends Over Time 



Category Trend 




Choose a category far details: 

• Likes US Pop Culture 12% 

• + Opinion of US Society/Cha.14% 

• America the Beautiful/Tourism 7% 

• Likes American Products, In... 6% 

• Dislikes US Pop Culture 6% 

• - Opinion of US Society/Cha... 9% 

• - View of American Gov't 11% 

• Facts 17% 

• Mixed Opinion 8% 
Teasing America 10% 



crimson hexagon 



3/1 0/1 D 



Show Legend 




4/2/10 



4/25/10 



5/18/11 
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Findings to Date 

• Russia 

• Conversations divided by politics (Nationalist, Democratic 
Opposition) 

• Topic: America and America's Society 

• Indonesia 

• Conversations cluster around Fashion, Celebrity, Culture 
and Public Discourse 

• Topic: Education in America 

• Pakistan 

• Low volume in Urdu, expanding universe to London 
diaspora 

• Topic: U.S. role in the Region 
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Evaluation & measurement Unit 




Goal 2: Assess Posts' processes for communicating with foreign 
publics over the Internet given local constraints 



Chris Toppe, Ph.D. 
Senior Evaluation Officer 
R/PPR EMU 
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evaluation & < i ln i ^1. Logic Model: Goal 2 





Inputs 


Process 


Process Outcomes 


Ultimate Outcomes 




Web Conversations about 
Topics of Interest to the DoS 


1. Conversation Monitoring 
across 10 topics in six languages 
(plus English) 

2. Classification of conversations 
according to tone and sentiment 
(initially in English only) 

3. Tracking of topic interest 
around planned events (pre- and 
post-test model) 


1. Tracking topical conversations over time 

2. Measurement of the impact of events on 
conversational quantity, tone, and sentiment 

3. Measurements of the influence of IIP 
interventions in influencing conversational 
quantity, tone, and sentiment 


1. Ability to focus IIP outreach activities when and 
where needed 

2. Ability to identify how conversations about DoS 
issues changes over time 


arch 










\ Resei 




1. Web survey of 


1. User profiles 


1. Identification of preferred Web 


Adaption and 


Internet users 


2. Web 2.0 tool usage 


2.0 interaction modes by sub-group 


i 


usage of Web 2.0 


2. Tracking of site 


3. Interest in US topics 


2. Identification of topic interests by 




applications across 


visits, information 


4. Engagement in US-provided 


sub-group 




and between 


views, and downloads 


media 


3. Measurement of success of IIP in 




countries (8) and 


from US websites 


5. Success of IIP to influence 


influencing usage of Us-provided 




cultures (several) 


3. Intentional and 
scheduled IIP 
interventions 


knowledge-seeking behavior 


material 




In-person 


TBD 


TBD 


TBD 




interviews in of 










Internet users in 










Pakistan 








Formative 


1. IIP Mission, Goals, and 
Objectives 

2. Review of IIP policies and 
procedures 

3. Review of IIP products and 
services, strategies and 
tactics 


1. Interviews, Document Review 

2. Compile policies and 
procedures 

3. Compile product and services 
inventory 

4. Document IIP strategies and 
tactics 


1. Unified statement of Mission, Goals and 
Objectives 

2. Measurement Needs and Metrics 

3. Policy review and recommendations 

4. Identification of Topics of Interest 

5. Comparison of products and services to industry 
standards 

6. Assessment of strategies and tactics in relation to 
goals and objectives 


1. Assessment of the relationships between mission, 
goals and objectives AND policies and procedures 
AND products and services AND strategies and tactics 

2. Recommendations for refinements in any of the 
above 

3. Implementation of a measurement methodology 

4. Measurement over time of changes 
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Evaluation & measurement Unit 



Outcomes 




• Advise posts on how to use web 2.0 tools and technologies to 
communicate with a broader, younger audience 

• Advise posts on entry points with this new audience 

- Issues 

- Topics 



Evaluation 

In- Country Assessments 

Research Questions 



How are Posts using Web 2.0 
tools? How are they integrated 
into a PD strategy? 

How are Posts handling content 
management, messaging, 
translations? What are Posts 
creating on their own? What are 
they leveraging from IIP? 

How are Posts dealing with 
infrastructure issues, such as 
Internet availability, accessibility, 
affordability, and issues including 
government monitoring, 
censorship, and laws restricting 
Internet usage/communications? 



Research Methodologies 



Key stakeholder (Regional Bureau and 
Post) dialogues via DVC; in-person 
interviews at Posts 

Focus groups with key foreign audiences 

Analysis of Web 2.0 physical and legal 
infrastructure, nationally and regionally 



Evaluation . 

Audiences: Focus Groups 



Key Research Questions 

What are Internet user 
demographics by country? What is 
the penetration of Web 2.0 
platforms? 



What Web 2.0 tools and platforms 
are they using? How do users seek, 
obtain and interpret knowledge 
from the internet? 



How are they sharing information 
discussed in these platforms? What 
is the ripple effect within and 
among web communities? 



and Surveys 



Research Methodologies 



Web 2.0 platform matrix and data 
dictionary: Regional Bureaus, Posts and 
field observation 



Survey in local language of 1,000 Internet 
users in each of the 6 participating 
countries, vetted with Post 



500 in-person interviews in Pakistan , 
Zambia, and Algeria 

PD Strategic Communications Plan 
metrics for Pakistan 



Evaluation Hot 

/ 

Focus Groups 

• Young adult Internet users 

• Two focus groups per country 

• Topics 

— How they use the Internet and Web 2.0 tools 

- What technologies they use and how they use them (cell 
phones as an example) 

— Their interest in international issues, especially as they relate 
to US issues 

- Their attitudes toward the US on key topics 

• Country specific 



Evaluation Hot 

/ 

Audience Surveys 

• Internet surveys of 1 ,000 people in 6 countries 

- Recruited from a panel 

- Representative of in-country internet users 

• In-person interviews in 3 countries to include internet and SMS 
users 

• Covers a variety of topics 

- Internet usage 

- Topic interests 

- Opinion of and attitudes toward the US 
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Findings to Date 

• Social media are social 

- Little use (so far) for issue discussions 

• Little interest in broad international issues 

- Topic discussions lack salience 

- They don't trust internet postings 

• Findings suggests posts will have to reach out first in social 
ways 



Evaluation & measurement Unit 




Goal 3: Provide recommendations for process improvements at 

Post and IIP level 



Dean Olsen 
Social Science Analyst 
R/PPR EMU 
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evaluation 1 ft) Logic Model: Goal 3 





Inputs 


Process 


Process Outcomes 


Ultimate Outcomes 


ket Research 


Web Conversations about 
Topics of Interest to the DoS 


1. Conversation Monitoring 
across 10 topics in six languages 
(plus English) 

2. Classification of conversations 
according to tone and sentiment 
(initially in English only) 

3. Tracking of topic interest 

dlUUIIU pidlllltJU fcJVfcJlllb ^pit;- dllU 

post-test model) 


1. Tracking topical conversations over time 

2. Measurement of the impact of events on 
conversational quantity, tone, and sentiment 

3. Measurements of the influence of IIP 
interventions in influencing conversational 
quantity, tone, and sentiment 


1. Ability to focus IIP outreach activities when and 
where needed 

2. Ability to identify how conversations about DoS 
issues changes over time 


Man 


Adaption and usage of Web 
2.0 applications across and 
between countries (8) and 
cultures (several) 


1. Web survey of Internet users 

2. Tracking of site visits, 
information views, and 
downloads from US websites 

3. Intentional and scheduled IIP 
interventions 


1. User profiles 

2. Web 2.0 tool usage 

3. Interest in US topics 

4. Engagement in US-provided media 

5. Success of IIP to influence knowledge-seeking 
behavior 


1. Identification of preferred Web 2.0 interaction 
modes by sub-group 

2. Identification of topic interests by sub-group 

3. Measurement of success of IIP in influencing usage 
of Us-provided material 




In-person interviews in of 
Internet users in Pakistan 


TBD 


TBD 


TBD 


Formative Evaluation 


1. IIP Mission, 
Goals, and 
Objectives 

2. Review of IIP 
policies and 
procedures 

3. Review of IIP 
products and 
services, strategies 
and tactics 


1. Interviews, 
Document Review 

2. Compile policies 
and procedures 

3. Compile product 
and services 
inventory 

4. Document IIP 
strategies and tactics 


1. Unified statement of Mission, 
Goals and Objectives 

2. Measurement Needs and 
Metrics 

3. Policy review and 
recommendations 

4. Identification of Topics of 
Interest 

5. Comparison of products and 
services to industry standards 

6. Assessment of strategies and 
tactics in relation to goals and 
objectives 


1. Assessment of the relationships 
between mission, goals and 
objectives AND policies and 
procedures AND products and 
services AND strategies and tactics 

2. Recommendations for 
refinements in any of the above 

3. Implementation of a 
measurement methodology 

4. Measurement over time of 
changes 
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EVALUATION & SUREMENT 

IIP Formative Evaluation 



Key Research Questions 



What tools and platforms are IIP 
using? How does their tool 
adoption compare to industry best 
practices? 



How does IIP measure success on 
social media platforms? What 
metrics are being used by IIP? How 
does these measurement practices 
compare to industry best practices? 
Are there things they would like to 
measure? What does IIP want to 
learn from the data? 



How does IIP work with Posts? 
What kind of feedback do they get 
from Posts? 



Research Methodologies 

Key stakeholder interviews 
(Washington) 

Create IIP electronic engagement 

platform matrix 

Best practices comparisons 

Compilation of Web 2.0 metrics 
being used by IIP 
Standard analysis of page views, 
downloads, foreign audience 
reached, and time spent on Post 
web sites and other platforms 
Identification of new metrics 
Best practices comparisons 



Washington and Post key 
stakeholder interviews 
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GOAL! Soccer for a Better Future 




Evaluation & measurement Unit 



IIP Social Media Platforms and Metrics 



Platform selection and usage 
Web usage metrics and data points 
Identification of additional measurement needs 



@ America.gov BKs * ■ 
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^ <fr D Embajada de Estados Unidos en Costa Rica | Facebook 





Dia de la 
Tierra 

Created about a 
month ago 



EARTH Day 
2010 

Created about a 
month ago 



Visit With Embassy Staff 
and Their Families [HQ] 

12:19 =: =: 



Secretary Clinton Press 

Conference 

15:21 



Create a Page for My Business 
Report Page 



igS Embajada de Estados Unidos en Costa Rica Yes, it is 
i^HE posted here in our FB page 

" May 25 at 1:01pm 

Embajada de Estados Unidos en Costa Rica La agricultura del siglo XXI 
La brega mas diffcil para la humanidad ha sido la lucha por alimentarse. Esta 
edicion de eJournal USA explora cdmo las proezas tecnicas del siglo XXI y habilidad 
en la agricultura son la clave para alimentar a las crecientes poblaciones del future 

Periodicos electronicos eJournal USA - PerkSdicos electronkros - America.gov 

www . america . gov 

?? 

£J May 25 at 11:25am Comment Like 

Embajada de Estados Unidos en Costa Rica 
21st-century Agriculture - America.gov 

www.america.gov 

Humanity's longest struggle has been to feed itself. Thise Journal USA explores how 21st- 
:entury technical prowess and agriculture ' 
jfthe future. Cover: ... 

0 May 25 at 11:21am Comment Like 



Embajada de Estados Unidos en Costa Rica 

Caribbean Basin Security Initiative 

www.america.gov 

The State Department releases a fact sheet that outlines key objectives of the Caribbean Basin 
Security Initiative, a shared regional security partnership between the United States and the 
Caribbean Basin. 

May 25 at 10:34am ■ Comment Like 
tl Jerry Carter likes this. 

Embajada de Estados Unidos en Costa Rica 
Iniciativa de SegurkJad de la Cuenca del Caribe 



Evaluation Sqr 

/ 

Process Outcomes 

• Recommendation for improving performance measurement and 
adopting new performance metrics 

• Identification of topics of Interest 

• Comparison of product usage and platform selection to industry 
standards 

• Assessment of strategies and tactics in relation to Posts' foreign 
audience engagement efforts 



Evaluation Hot 
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Ultimate Outcomes 

• Implementation of a measurement methodology and data 
collection process 

• Improved messaging and content management 

• Improved product review and adoption process 

• Optimization of IIP Web 2.0 initiatives with Posts' Web 2.0 
initiatives 

• Increase in key foreign audiences reached by HP's electronic 
media engagement efforts 



Evaluation & measurement Unit 




Measuring Public Diplomacy in the 21 st Century 



The 21 st century electronic media communication environment 
has unique challenges and opportunities, our mission is to help 
IIP identify and seize communication opportunities and mitigate 
challenges with innovative evaluation tools and approaches that 
match a changing global landscape of engagement. 



EVALUATION & MEASUREMENT UNIT 




Concluding Remarks 

Cherreka Montgomery 
Director, R/PPR EMU 
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Evaluation Hot 

/ 

Three Phases 

• Market research 

- Web conversation analysis 

- Who is saying what about topics that interest us? 

- What is the tone of those conversations and does that tone 
change? 

• In-country assessment 

- Who is doing what with Web 2.0 tools? 

- What are the opportunities for post engagement? 

• Process improvements 

- How can IIP better serve posts? 

- How can IIP better measure what it does? 



